Apparatus and method for measuring position of stereo camera

ABSTRACT

An apparatus and method for measuring the position of a stereo camera. The apparatus for measuring a position of the camera according to an embodiment includes a feature point extraction unit for extracting feature points from images captured by a first camera and a second camera and generating a first feature point list based on the feature points, a feature point recognition unit for extracting feature points from images captured by the cameras after the cameras have moved, generating a second feature point list based on the feature points, and recognizing actual feature points based on the first feature point list and the second feature point list, and a position variation measurement unit for measuring variation in positions of the cameras based on variation in relative positions of the actual feature points.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/046,929, filed Feb. 18, 2016, which claims priority to Korean PatentApplication No. 10-2015-0091444, filed Jun. 26, 2015, which are herebyincorporated by reference in their entirety into this application.

BACKGROUND OF THE INVENTION 1. Technical Field

The present invention relates generally to technology for measuring theposition of a stereo camera composed of two cameras in real time usingfeature points installed on an object.

2. Description of the Related Art

Technologies for calculating in real time the position of a stereocamera by observing feature points present in space using the stereocamera have been widely used in the academic world and the industrialworld, including a three-dimensional (3D) scanner, a mobile robot, and ahuman-computer interface. Among these technologies, technology forcalculating the position of a camera using randomly arranged featurepoints without previously knowing the positions of the feature pointsmay be regarded as being highly useful due to the convenience ofconstruction. Among these technologies, technology that requires atexture around feature points, which enables the feature points insequentially input images to be distinguished from each other, may beutilized when feature points that are present in the texture of theoriginal background are used, without requiring feature points to beintentionally arranged. However, when there is an insufficient textureand feature points arranged by a user are used, there is inconveniencein which an identifiable texture or individual identification markersmust be additionally arranged around the feature points. In conventionaltechnology, there is a method for detecting, as the characteristics ofonly respective feature points, distances to neighboring feature pointsbased on the 3D positions of the feature points, which are reconstructedvia triangulation using stereo images, thus distinguishing the featurepoints. However, there may occur the case where it is ambiguous todistinguish such feature points using only this method.

U.S. Pat. No. 7,925,049 discloses technology for receiving images of astereo camera, distinguishing feature points from each other using thebackground texture of the images, and tracking the images.

However, the technology of U.S. Pat. No. 7,925,049 is fatallydisadvantageous in that it is impossible to precisely distinguishfeature points unless a background texture is present.

Therefore, there is a need for technology that precisely distinguishesfeature points from each other without using a texture or the like,present in a background, and measures the position of the camera basedon the feature points.

SUMMARY OF THE INVENTION

Accordingly, the present invention has been made keeping in mind theabove problems occurring in the prior art, and an object of the presentinvention is to measure the position of a stereo camera based on thepositions of feature points present in multiple frames.

Another object of the present invention is to measure variation in theposition of a camera using only feature points without requiring atexture around feature points or individual identification marks.

A further object of the present invention is to simply measure variationin the position of a camera, thus enabling the measurement of positionvariation to be easily applied to 3D object scanning devices.

In accordance with an aspect of the present invention to accomplish theabove objects, there is provided an apparatus for measuring a positionof a camera, including a feature point extraction unit for extractingfeature points from images respectively captured by a first camera and asecond camera and generating a first feature point list based on thefeature points; a feature point recognition unit for extracting featurepoints from images captured by the cameras after the cameras have moved,generating a second feature point list based on the feature points, andrecognizing actual feature points based on the first feature point listand the second feature point list; and a position variation measurementunit for measuring variation in positions of the cameras based on theactual feature points.

The feature point extraction unit may include an epipolar linegeneration unit for generating epipolar lines corresponding torespective feature points in the image captured by the first camera; andan extraction unit for extracting feature points corresponding to theepipolar lines from the image captured by the second camera.

The extraction unit may measure distances from the feature points in theimage captured by the second camera to the epipolar lines, and extractsfeature points, from which distances to the epipolar lines are less thanor equal to a specific distance.

The feature point recognition unit may include a coordinate extractionunit for extracting 3D position coordinates corresponding to respectivefeature points present in the first feature point list and the secondfeature point list; a distance measurement unit for measuring distancesbetween the feature points based on the 3D position coordinates; and arecognition unit for searching for feature points present in the secondfeature point list that are identical to the feature points present inthe first feature point list based on the measured distances, andrecognizing found feature points as actual feature points.

The recognition unit may be configured to, if feature points, from whichdistances to the feature points present in the first feature point listare less than or equal to a specific distance, are present in the secondfeature point list, recognize the feature points present in the firstfeature point list as the actual feature points.

The recognition unit may recognize the actual feature points usingRandom Sample Consensus (RANSAC).

The recognition unit may include a rigid transformation calculation unitfor calculating rigid transformations of respective feature pointspresent in the first feature point list, based on the feature pointspresent in the first feature point list and feature points present inthe second feature point list that are located at a distance less thanor equal to a specific distance from the feature points present in thefirst feature point list; and a feature point recognition unit forrecognizing the actual feature points based on the rigidtransformations.

The feature point recognition unit may recognize feature points thathave highest consistency in rigid transformations and are present in thesecond feature point list as actual feature points.

The position measurement unit may include a calculation unit forcalculating a rotation matrix and a translation vector based on theactual feature points; a rigid transformation calculation unit forcalculating rigid transformations based on the rotation matrix and thetranslation vector; and a position variation measurement unit formeasuring variation in positions of the first and second cameras basedon the rigid transformations.

In accordance with another aspect of the present invention to accomplishthe above objects, there is provided a method for measuring a positionof a camera, including extracting feature points from imagesrespectively captured by a first camera and a second camera andgenerating a first feature point list based on the feature points;extracting feature points from images captured by the cameras after thecameras have moved, generating a second feature point list, andextracting actual feature points based on the first feature point listand the second feature point list; and measuring variation in positionsof the cameras based on the actual feature points.

Generating the first feature point list may include generating epipolarlines corresponding to respective feature points in the image capturedby the first camera; and extracting feature points corresponding to theepipolar lines from the image captured by the second camera.

Generating the first feature point list may include measuring distancesfrom the feature points in the image captured by the second camera tothe epipolar lines, and extracting feature points, from which distancesto the epipolar lines are less than or equal to a specific distance.

Recognizing the actual feature points may include extracting 3D positioncoordinates corresponding to respective feature points present in thefirst feature point list and the second feature point list; measuringdistances between the feature points based on the 3D positioncoordinates; and searching for feature points present in the secondfeature point list that are identical to the feature points present inthe first feature point list based on the measured distances, andrecognizing found feature points as actual feature points.

Recognizing the actual feature points may be configured to, if featurepoints, from which distances to the feature points present in the firstfeature point list are less than or equal to a specific distance, arepresent in the second feature point list, recognize the feature pointshaving the distances less than or equal to the specific distance as theactual feature points.

Recognizing the actual feature points may be configured to recognize theactual feature points using Random Sample Consensus (RANSAC).

Recognizing the actual feature points may include calculating rigidtransformations of respective feature points present in the firstfeature point list, based on the feature points present in the firstfeature point list and feature points present in the second featurepoint list that are located at a distance less than or equal to aspecific distance from the feature points present in the first featurepoint list; and determining actual feature points based on the rigidtransformations.

Recognizing the actual feature points may be configured to recognizefeature points that have highest consistency in rigid transformationsand are present in the second feature point list as actual featurepoints.

Measuring the variation in the positions of the first camera and thesecond camera may include calculating a rotation matrix and atranslation vector based on the actual feature points; calculating rigidtransformations based on the rotation matrix and the translation vector;and measuring variation in positions of the first and second camerasbased on the rigid transformations.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will be more clearly understood from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a block diagram showing an apparatus for measuring theposition of a camera according to an embodiment of the presentinvention;

FIG. 2 is a block diagram showing the feature point extraction unitshown in FIG. 1;

FIG. 3 is a block diagram showing the feature point recognition unitshown in FIG. 1;

FIG. 4 is a block diagram showing the position variation measurementunit shown in FIG. 1;

FIGS. 5 and 6 are diagrams showing feature points used by the cameraposition measurement apparatus according to an embodiment of the presentinvention;

FIGS. 7 and 8 are diagrams showing the procedure in which the featurepoint extraction unit shown in FIG. 1 extracts feature points usingepipolar lines;

FIG. 9 is a diagram showing feature points measured by the cameraposition measurement apparatus while moving according to an embodimentof the present invention;

FIGS. 10 and 11 are diagrams showing applications of the apparatus formeasuring the position of a camera according to an embodiment of thepresent invention;

FIG. 12 is a diagram showing the scanning of an object using the cameraposition measurement apparatus according to an embodiment of the presentinvention;

FIG. 13 is an operation flowchart showing a method for measuring theposition of a camera according to an embodiment of the presentinvention;

FIG. 14 is an operation flowchart showing in detail the step ofcalculating rigid transformations based on actual feature points, shownin FIG. 13; and

FIG. 15 is an embodiment of the present invention implemented in acomputer system.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will be described in detail below with referenceto the accompanying drawings. Repeated descriptions and descriptions ofknown functions and configurations which have been deemed to make thegist of the present invention unnecessarily obscure will be omittedbelow. The embodiments of the present invention are intended to fullydescribe the present invention to a person having ordinary knowledge inthe art to which the present invention pertains. Accordingly, theshapes, sizes, etc. of components in the drawings may be exaggerated tomake the description clearer.

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the attached drawings.

FIG. 1 is a block diagram showing an apparatus for measuring theposition of a camera according to an embodiment of the presentinvention.

Referring to FIG. 1, the camera position measurement apparatus accordingto the embodiment of the present invention includes a feature pointextraction unit 110, a feature point recognition unit 120, and aposition variation measurement unit 130.

The feature point extraction unit 110 extracts feature points fromimages respectively captured by a first camera and a second camera, andgenerates a first feature point list.

Here, the first camera and the second camera may be cameras providedinside the camera position measurement apparatus.

The feature points may be specific points having a predetermined area sothat the feature points are extracted from the images.

In this case, the feature points may be specific points attached to anobject in the images captured by the cameras. For example, the featurepoints are circular figures attached to the object shown in FIGS. 5 to8.

Here, the shapes of feature points are not especially limited. Further,the shapes of feature points may be identical to each other, but mayalso be different from each other. When conventional technology is used,it is difficult to measure the position of the camera when the shapes offeature points are identical to each other, but the present inventionmay be used to easily measure the position of the camera even if theshapes of the feature points are identical to each other.

Here, the first feature point list may denote a set of feature pointswhich are present in an image captured by the second camera and whichcorrespond to one of the feature points in the image captured by thefirst camera.

For the first feature point list, a description will be made withreference to FIGS. 7 and 8. When feature points intersecting epipolarlines among the feature points extracted in FIG. 8 are PR, a featurepoint pair (PL, PR) is added to a corresponding point candidate list.Actually, the corresponding feature points do not exactly intersectepipolar lines due to a correction error and a feature point extractionerror. Accordingly, when the number of feature points falling within therange of a specific distance threshold on the epipolar lines is assumedto be M, a list of feature point pairs (PL, PRi) may be generated forall feature points PRi, where i=1 to M, and a set of feature point pairsmay be the first feature point list.

Here, when the first feature point list is generated, a first featurepoint list including all feature points in the image captured by thefirst camera may be generated.

The feature points in the image captured by the second camera, whichcorrespond to the feature points in the image captured by the firstcamera, may generate epipolar lines using the feature points in theimage captured by the first camera, and may denote the feature points inthe image captured by the second camera, which correspond to theepipolar lines.

Here, the feature points in the image captured by the second camera,which correspond to the epipolar lines, may also denote feature pointsintersecting the epipolar lines.

Since the feature points may not intersect the epipolar lines due to theinternal correction error and the feature point extraction error of thecameras, feature points that do not intersect the epipolar lines, buthave distances to the epipolar lines, less than or equal to a specificdistance, may be recognized as feature points corresponding to thefeature points in the image captured by the first camera.

The feature point recognition unit 120 moves the cameras, extractsfeature points from images captured by the moved cameras, generates asecond feature point list, and recognizes actual feature points based onthe first feature point list and the second feature point list.

Here, the method of generating the second feature point list isidentical to the method of generating the first feature point list.

The second feature point list may be generated using the images capturedby the cameras after the cameras have moved.

The feature points included in the second feature point list may beidentical to those included in the first feature point list, may includenew additional feature points, or may omit feature points that wereincluded in the first feature point list. The reason for this is thatthe images are captured by the cameras while the cameras are moving, andthus an image captured before movement is different from an imagecaptured after movement. In greater detail, when each camera capturesimages at different angles while moving, feature points that could notbe captured in a previous image may be captured in a current image, andfeature points that were captured in a previous image may not becaptured by the lens on the camera when viewed from another angle.

In this case, the actual feature points may be feature points that arepresent both in the first feature point list and in the second featurepoint list.

Here, when the actual feature points are recognized, the feature pointspresent in the first feature point list and the second feature pointlist may be extracted as 3D position coordinates, respectively. Whenattempting to extract 3D position coordinates, the 3D positioncoordinates may be extracted based on the first camera or the secondcamera.

In this case, the distances between feature points may be measured basedon the extracted 3D position coordinates, and the feature points in thesecond feature point list that correspond to feature points in the firstfeature point list may be found based on the measured distances. Thefeature points in the second feature point list may be extracted as theactual feature points.

The feature points, the distances between which are calculated as valuesless than or equal to a specific distance value, may be extracted as theactual feature points.

Here, when feature points are extracted as the actual feature points,error may be reduced using Random Sample Consensus (RANSAC).

When the actual feature points are extracted, recent contact pointsamong the feature points of the second feature point list are obtainedfor respective feature points of the first feature point list, rigidtransformations are calculated for the recent contact points, and actualfeature points may be extracted based on whether the rigidtransformations have the same transformation relationship between thecontact points.

The position variation measurement unit 130 measures variation in thepositions of the cameras based on the actual feature points.

A detailed description of the position variation measurement unit 130will be made later with reference to FIG. 4.

FIG. 2 is a block diagram showing the feature point extraction unitshown in FIG. 1.

Referring to FIG. 2, the feature point extraction unit 110 includes anepipolar line generation unit 210 and an extraction unit 220.

The epipolar line generation unit 210 generates epipolar linescorresponding to respective feature points in an image captured by thefirst camera.

Epipolar lines denote lines passing through epipoles in stereo visionfields. Since a description thereof is well known to those skilled inthe art to which the present invention pertains, it will be omittedhere.

The extraction unit 220 extracts feature points corresponding to theepipolar line from an image captured by the second camera.

Here, the extraction unit 220 measures distances between the featurepoints in the image captured by the second camera and the epipolarlines, and may extract feature points, from which the distances to theepipolar lines are less than or equal to a specific distance, as featurepoints.

When there are multiple feature points in the image captured by thefirst camera, the extraction unit 220 may perform a task of extractingfeature points based on measured distances between the epipolar linescorresponding to respective feature points and the feature points in theimage captured by the second camera.

FIG. 3 is a block diagram showing the feature point recognition unit 120shown in FIG. 1.

Referring to FIG. 3, the feature point recognition unit 120 includes acoordinate extraction unit 310, a distance measurement unit 320, and arecognition unit 330.

The coordinate extraction unit 310 extracts 3D position coordinates ofrespective feature points present in a first feature point list and in asecond feature point list.

The distance measurement unit 320 measures the distances between thefeature points present in the first feature point list and the featurepoints present in the second feature point list, based on the 3Dposition coordinates.

Here, the measurement of the distances means that the distance betweenany one of the feature points present in the first feature point listand any one of the feature points present in the second feature pointlist, rather than the distance between the feature points present in thefirst feature point list or the distance between the feature pointspresent in the second feature point list, is measured.

The recognition unit 330 individually searches for feature points thatare present in the second feature point list and are identical to thefeature points present in the first feature point list, based on themeasured distances, and recognizes the identical feature points asactual feature points if the identical feature points are found in thesecond feature point list.

Here, the actual feature points may be feature points that are presentboth in the first feature point list and the second feature point list.

When the actual feature points are recognized, the feature pointspresent in the first feature point list and the second feature pointlist may be extracted as respective 3D position coordinates. When 3Dposition coordinates are extracted, they may be extracted based oneither the first camera or the second camera.

Here, the distances between feature points may be measured based on theextracted 3D position coordinates. Feature points in the second featurepoint list, which correspond to the feature points in the first featurepoint list, may be found based on the measured distances, and thefeature points present in the second feature point list may be extractedas actual feature points.

The feature points, the distances between which are calculated as beingless than or equal to a specific value, may be extracted as actualfeature points.

In this case, the actual feature points may also be extracted in such away as to generate multiple frames by performing capturing using thefirst camera and the second camera, extract actual feature pointsbetween respective frames, assign numbers to the actual feature points,and assign the same number to the same feature points. In this case, theactual feature points are extracted using multiple frames, and thusprecision is improved.

FIG. 4 is a block diagram showing the position variation measurementunit shown in FIG. 1.

Referring to FIG. 4, the position variation measurement unit 130includes a rigid transformation calculation unit 410 and a positionvariation calculation unit 420.

The rigid transformation calculation unit 410 calculates a rotationmatrix and a translation vector, based on extracted actual featurepoints.

Here, the rotation matrix and the translation vector may be calculatedby comparing a reference frame with the current frame.

A start frame, in which an image is initially input to a stereo device,may be set as a reference frame. The rotation matrix and the translationvector of the coordinate system of the start frame may be set as a unitmatrix and a zero vector, respectively.

There is no limitation as to the method for setting the reference frame.However, when the number of identical feature points between the currentframe and the reference frame is found to be equal to or greater than aspecific number by comparing the number of identical feature pointsbetween the current frame and the reference frame, the current frame isset as the reference frame, and then a rotation matrix and a translationvector may be calculated.

Here, when the number of actual feature points that are identicalbetween the reference frame and the current frame is less than thespecific number, a subsequent frame is set as the reference frame.Further, the number of actual feature points that are identical mayagain be compared with the specific number.

The specific number is not especially limited. It is preferable that thespecific number is equal to or greater than 3, but it is not limitedthereto.

Here, the scheme for calculating the rigid transformations between thereference frame and the current frame is not especially limited. Forexample, the three points that are farthest away from a 3D point in thereference frame are obtained. The center of gravity of the three pointsis calculated and is designated as c0. Then, vectors from the center ofgravity to the three points are designated as p0, q0, and r0,respectively. x0, y0, and z0 are obtained using the following Equation(1):

x0=p0/∥p0∥

b0=q0/∥q0∥

z0=b0×x0

y=z0×x0  (1)

In this case, x1, y1, and z1 may be obtained for the current frame usingthe same method as that given in Equation (1).

Here, the rotation matrix (Re) and the translation vector (te) may beobtained using the following Equation (2):

Re=[x1y1z1][x0y0z0]⁻¹

te=−[x1y1z1][x0y0z0]⁻¹ c0+c1  (2)

The rotation matrix and the translation vector for the rigidtransformations may be stored and loaded for each frame.

The results of the rigid transformations obtained using Equation (2) maybe used as initial values of the nonlinear optimization calculation inorder to obtain more precise results.

The cost function of the nonlinear optimization calculation isrepresented by the following Equation (3):

$\begin{matrix}{\sum\limits_{i = 1}^{N}{{{RcXri} + {tc} - {Xci}}}^{2}} & (3)\end{matrix}$

where N denotes the number of 3D points common to both the referenceframe and the current frame, Rc denotes the rotation matrixcorresponding to the rigid transformations between the reference frameand the current frame, tc denotes the translation vector correspondingto the rigid transformation between the reference frame and the currentframe, Xri denotes the coordinates of 3D feature points, common to boththe reference frame and the current frame, in the reference frame, andXci denotes the coordinates of 3D feature points, common to both thereference frame and the current frame, in the current frame.

Here, for the nonlinear optimization procedure, any type of nonlinearoptimization method developed in optimization fields may be used.

Further, estimated values may be refined based on observed values, andthus more precise results may be obtained. With the exception of virtualpoints for which the succession of numbers does not continue for apredetermined frame or longer, the coordinates of remaining featurepoints on the image captured by the left camera and the image capturedby the right camera are used as the measured values. The reconstructedpoints in 3D space and the calculated positional values of the stereocamera for the respective frames are used as initial estimated values,after which more precise results may be extracted using bundleadjustment. Points in 3D space that have very short spatial distancesless than or equal to a predetermined threshold therebetween areregarded as identical points and are subjected to bundle adjustment,thus suppressing the drift of the points.

The position variation calculation unit 420 integrates the rigidtransformations obtained by the rigid transformation calculation unit410 with a rigid transformation composed of a rotation matrix and atranslation vector in the coordinate system of the reference frame, andthen obtains the current position of the stereo camera based on thecoordinate system of the start frame.

FIGS. 5 and 6 are diagrams showing feature points used by the cameraposition measurement apparatus according to an embodiment of the presentinvention.

FIG. 5 is a diagram showing an object captured by a first camera andFIG. 6 is a diagram showing an object captured by a second camera.

Feature points may be attached to the object. Referring to FIGS. 5 and6, it can be seen that feature points are attached to the outside of theobject.

Here, the first camera and the second camera may be cameras provided inthe camera position measurement apparatus.

The feature points may be specific points having a predetermined area sothat the feature points are extracted from the images.

In this case, the feature points may be specific points attached to anobject in the images captured by the cameras. For example, the featurepoints are circular figures attached to the object shown in FIGS. 5 to8.

Here, the shapes of feature points are not especially limited. Further,the shapes of feature points may be identical to each other, but mayalso be different from each other. When conventional technology is used,it is difficult to measure the position of the camera when the shapes offeature points are identical to each other, but the present inventionmay be used to easily measure the position of the camera even if theshapes of the feature points are identical to each other.

FIGS. 7 and 8 are diagrams showing the procedure in which the featurepoint extraction unit shown in FIG. 1 extracts feature points usingepipolar lines.

Referring to FIG. 7, it can be seen that numbers are assigned to featurepoints in an image captured by the first camera.

Referring to FIG. 8, it can be seen that the numbers of the featurepoints in the image captured by the first camera are assigned torespective feature points in the image captured by the second camera.

In this case, feature points in the captured image shown in FIG. 8 thatcorrespond to the feature points in the image shown in FIG. 7 maygenerate epipolar lines using the feature points in the image shown inFIG. 7, and may denote the feature points in the image shown in FIG. 8,which correspond to the epipolar lines.

Here, the feature points in the image shown in FIG. 8 that correspond tothe epipolar lines may also be feature points that intersect theepipolar lines.

Since the feature points may not intersect the epipolar lines due to theinternal correction error and the feature point extraction error of thecameras, feature points that do not intersect the epipolar lines, buthave distances to the epipolar lines, less than or equal to a specificdistance, may be recognized as feature points corresponding to thefeature points in the image shown in FIG. 7.

FIG. 9 is a diagram showing feature points measured by the cameraposition measurement apparatus while moving according to an embodimentof the present invention.

Referring to FIG. 9, camera position measurement apparatuses 901 to 903according to an embodiment of the present invention and feature points904 to 915 are illustrated.

The camera position measurement apparatus 901 indicates the position ofthe camera position measurement apparatus at time t, and the cameraposition measurement apparatus 902 indicates the position of the cameraposition measurement apparatus at time t+1. The camera positionmeasurement apparatus 903 illustrates the cases where the positions offeature points, measured when the measurement apparatuses 901 and 902are superimposed on each other, are measured differently. That is, thedrawing shows that, while the camera position measurement apparatus ismoving, the relative positions of the feature points 904 to 907 arevaried as the reference (camera position measurement apparatus) of arelative coordinate system is changed.

That is, since the feature points 908 and 909 shown on the right side ofFIG. 9 are located relatively close to each other, the feature points908 and 909 are considered to be the same feature point 904, and mayalso be regarded as an actual feature point. The remaining featurepoints 905 to 907 and 910 to 915 may be considered in the same manner.

FIGS. 10 to 12 are diagrams showing applications of the camera positionmeasurement apparatus according to embodiments of the present invention.

FIG. 10 illustrates a system in which the camera position measurementapparatus is applied and attached to a scanner, wherein the systemincludes a first camera 1010, a second camera 1020, and a laser 1030.

FIG. 11 is a diagram showing the state in which the camera positionmeasurement apparatus according to the embodiment of the presentinvention is applied to a 3D object scanner and scans an object. It canbe seen from FIG. 11 that the camera position measurement apparatusaccording to the embodiment of the present invention may measure theposition of the camera and then be applied to the 3D object scanner.

When feature markers are attached to an object corresponding to a scantarget, and the object is captured by a stereo camera while the stereocamera is moving, the stereo camera may function as a positionrecognition sensor.

Here, when a laser line scanner composed of a camera and a line laser ismounted, 3D scan information may be extracted for the laser line of eachframe. When a set of 3D points corresponding to such scan results iscalled X(t), and a positional value measured by the stereo camera isrepresented by a rotation matrix R(t) and a translation vector t(t), astart frame coordinate system Xw may be represented by the followingEquation (4):

Xw(t)=R(t)[RsX(t)+ts]+t(t)  (4)

where Rs denotes a rotation matrix indicating the coordinate conversionrelationship between the laser line scanner and the left camera of thestereo camera, and is denotes a translation vector indicating thecoordinate conversion relationship between the laser line scanner andthe left camera of the stereo camera.

FIG. 12 is a diagram showing the results obtained by applying the cameraposition measurement apparatus of the embodiment of the presentinvention to a 3D object scanner and then scanning an object.

FIG. 12 illustrates the results of scanning an object based on theresults of the combination of Xw(t) shown in FIG. 11 in space.

FIG. 13 is an operation flowchart showing the method for measuring theposition of a camera according to an embodiment of the presentinvention.

Referring to FIG. 13, corresponding feature points are extracted fromimages captured by a first camera and a second camera, and a firstfeature point list is generated at step S1310.

Here, the first camera and the second camera may be cameras providedinside the camera position measurement apparatus.

The feature points may be specific points having a predetermined area sothat the feature points are extracted from the images.

In this case, the feature points may be specific points attached to anobject in the images captured by the cameras. For example, the featurepoints are circular figures attached to the object shown in FIGS. 5 to8.

Here, the shapes of feature points are not especially limited. Further,the shapes of feature points may be identical to each other, but mayalso be different from each other. When conventional technology is used,it is difficult to measure the position of the camera when the shapes offeature points are identical to each other, but the present inventionmay be used to easily measure the position of the camera even if theshapes of the feature points are identical to each other.

Here, the first feature point list may denote a set of feature pointswhich are present in an image captured by the second camera and whichcorrespond to one of the feature points in the image captured by thefirst camera.

For the first feature point list, a description will be made withreference to FIGS. 7 and 8. When feature points intersecting epipolarlines among the feature points extracted in FIG. 8 are PR, a featurepoint pair (PL, PR) is added to a corresponding point candidate list.Actually, the corresponding feature points do not exactly intersectepipolar lines due to a correction error and a feature point extractionerror. Accordingly, when the number of feature points falling within therange of a specific distance threshold on the epipolar lines is assumedto be M, a list of feature point pairs (PL, PRi) may be generated forall feature points PRi, where i=1 to M, and a set of feature point pairsmay be the first feature point list.

Here, when the first feature point list is generated, a first featurepoint list including all feature points in the image captured by thefirst camera may be generated.

The feature points in the image captured by the second camera, whichcorrespond to the feature points in the image captured by the firstcamera, may generate epipolar lines using the feature points in theimage captured by the first camera, and may denote the feature points inthe image captured by the second camera, which correspond to theepipolar lines.

Here, the feature points in the image captured by the second camera,which correspond to the epipolar lines, may also denote feature pointsintersecting the epipolar lines.

Since the feature points may not intersect the epipolar lines due to theinternal correction error and the feature point extraction error of thecameras, feature points that do not intersect the epipolar lines, buthave distances to the epipolar lines, less than or equal to a specificdistance, may be recognized as feature points corresponding to thefeature points in the image captured by the first camera.

Further, corresponding feature points are extracted from images capturedagain by the cameras after the cameras have moved, and a second featurepoint list is generated at step S1320.

Here, the method of generating the second feature point list isidentical to the method of generating the first feature point list.

The second feature point list may be generated using the images capturedby the cameras after the cameras have moved.

The feature points included in the second feature point list may beidentical to those included in the first feature point list, may includenew additional feature points, or may omit feature points that wereincluded in the first feature point list. The reason for this is thatthe images are captured by the cameras while the cameras are moving, andthus an image captured before movement is different from an imagecaptured after movement. In greater detail, when each camera capturesimages at different angles while moving, feature points that could notbe captured in a previous image may be captured in a current image, andfeature points that were captured in a previous image may not becaptured by the lens on the camera when viewed from another angle.

Further, actual feature points are extracted at step S1330.

In this case, the actual feature points may be feature points that arepresent both in the first feature point list and in the second featurepoint list.

Here, when the actual feature points are recognized, the feature pointspresent in the first feature point list and the second feature pointlist may be extracted as 3D position coordinates, respectively. Whenattempting to extract 3D position coordinates, the 3D positioncoordinates may be extracted based on the first camera or the secondcamera.

In this case, the distances between feature points may be measured basedon the extracted 3D position coordinates, and the feature points in thesecond feature point list that correspond to feature points in the firstfeature point list may be found based on the measured distances. Thefeature points in the second feature point list may be extracted as theactual feature points.

The feature points, the distances between which are calculated as valuesless than or equal to a specific distance value, may be extracted as theactual feature points.

Here, when feature points are extracted as the actual feature points,error may be reduced using Random Sample Consensus (RANSAC).

When the actual feature points are extracted, recent contact pointsamong the feature points of the second feature point list are obtainedfor respective feature points of the first feature point list, rigidtransformations are calculated for the recent contact points, and actualfeature points may be extracted based on whether the rigidtransformations have the same transformation relationship between thecontact points.

Further, rigid transformations are calculated based on the actualfeature points at step S1340.

Here, a rotation matrix and a translation vector are calculated based onthe extracted actual feature points.

The rotation matrix and the translation vector may be calculated bycomparing a reference frame with a current frame.

There is no limitation as to the method for setting the reference frame.However, when the number of identical feature points between the currentframe and the reference frame is found to be equal to or greater than aspecific number by comparing the number of identical feature pointsbetween the current frame and the reference frame, the current frame isset as the reference frame, and then a rotation matrix and a translationvector may be calculated.

Here, when the number of actual feature points that are identicalbetween the reference frame and the current frame is less than thespecific number, a subsequent frame is set as the reference frame.Further, the number of actual feature points that are identical mayagain be compared with the specific number.

The specific number is not especially limited. It is preferable that thespecific number is equal to or greater than 3, but it is not limitedthereto.

Here, the scheme for calculating the rigid transformations between thereference frame and the current frame is not especially limited. Forexample, the three points that are farthest away from a 3D point in thereference frame are obtained. The center of gravity of the three pointsis calculated and is designated as c0. Then, vectors from the center ofgravity to the three points are designated as p0, q0, and r0,respectively. x0, y0, and z0 are obtained using the following Equation(1):

x0=p0/∥p0∥

b0=q0/∥q0∥

z0=b0×x0

y=z0×x0  (1)

Next, for the current frame, x1, y1, and z1 may be obtained using thesame method as that given in Equation (1).

Finally, a rotation matrix (Re) and a translation vector (te) may beobtained using the following Equation (2).

Re=[x1y1z1][x0y0z0]⁻¹

te=−[x1y1z1][x0y0z0]⁻¹ c0+c1  (2)

Further, variation in the positions of the cameras is measured based onthe rigid transformations at step S1350.

FIG. 14 is an operation flowchart showing in detail the step ofcalculating rigid transformations based on the actual feature pointsshown in FIG. 13.

Referring to FIG. 14, numbers assigned to the feature points of areference frame and a current frame are compared with each other at stepS1410.

When the feature points present in the reference frame and the currentframe are identical to each other, the numbers assigned thereto are alsoidentical to each other.

Further, whether the number of identical numbers is equal to or greaterthan a specific number is determined at step S1420. If it is determinedthat the number of identical numbers is less than the specific number,the reference frame is changed at step S1430.

Here, the specific number is not especially limited. It is preferablethat the specific number is equal to or greater than 3, but it is notlimited thereto.

Further, when the number of identical numbers is equal to or greaterthan the specific number, rigid transformations between the referenceframe and the current frame are calculated at step S1440.

In this case, the rigid transformations may be calculated based on thevariation in the positions of the feature points using the identicalfeature points.

The method of calculating the rigid transformations has been describedin detail with reference to FIG. 13.

Further, the rigid transformations between the start frame and thecurrent frame are calculated at step S1450.

Here, the rigid transformations may be calculated based on the variationin the positions of feature points using the identical feature points.

Accordingly, the present invention may observe feature points using astereo camera, and simply measure the position of the camera based onthe positions of feature points present in multiple frames.

Further, the present invention measures the position of the camera viathe comparison between frames, thus conveniently measuring variation inthe position of the camera without requiring a texture around featurepoints or individual identification marks.

Furthermore, the present invention may simply measure variation in theposition of a camera, and then be easily applied to 3D object scanningdevices.

FIG. 15 is an embodiment of the present invention implemented in acomputer system.

Referring to FIG. 15, an embodiment of the present invention may beimplemented in a computer system, e.g., as a computer readable medium.As shown in in FIG. 15, a computer system 1520-1 may include one or moreof a processor 1521, a memory 1523, a user input device 1526, a useroutput device 1527, and a storage 1528, each of which communicatesthrough a bus 1522. The computer system 1520-1 may also include anetwork interface 1529 that is coupled to a network 1530. The processor1521 may be a central processing unit (CPU) or a semiconductor devicethat executes processing instructions stored in the memory 1523 and/orthe storage 1528. The memory 1523 and the storage 1528 may includevarious forms of volatile or non-volatile storage media. For example,the memory may include a read-only memory (ROM) 1524 and a random accessmemory (RAM) 1525.

Accordingly, an embodiment of the invention may be implemented as acomputer implemented method or as a non-transitory computer readablemedium with computer executable instructions stored thereon. In anembodiment, when executed by the processor, the computer readableinstructions may perform a method according to at least one aspect ofthe invention.

As described above, in the apparatus and method for measuring theposition of a stereo camera according to the present invention, theconfigurations and schemes in the above-described embodiments are notlimitedly applied, and some or all of the above embodiments can beselectively combined and configured so that various modifications arepossible.

What is claimed is:
 1. An apparatus for measuring a 3D position of anobject, comprising: a laser line scanner obtaining frame-by-frame scanresults for the object using a line laser and a single camera; a stereocamera measuring position variation information of the stereo camerausing images obtained from the stereo camera with respect to the object;and a measurement unit measuring 3D position information of the objectbased on the position variation information of the stereo camera and thescan results for the object.
 2. The apparatus of claim 1, wherein thestereo camera is configured to: extract feature points from imagescaptured by cameras included in the stereo camera and generate a firstfeature point list based on the feature points; extract feature pointsfrom images captured by the cameras after the cameras have moved,generate a second feature point list based on the feature points, andrecognize actual feature points based on the first feature point listand the second feature point list; and measure the position variationinformation of the stereo camera based on variations in relativepositions of the actual feature points.
 3. The apparatus of claim 2,wherein the measurement unit is configured to: convert the coordinatesof all the frames of the scan results into a coordinate system of areference frame based on the position variation information of thestereo camera and the positional relationship information between thestereo camera and the laser line scanner; and obtain a set of convertedcoordinates of all the frames as the 3D position information of theobject.
 4. The apparatus of claim 3, wherein the position variationinformation of the stereo camera includes a first rotation matrix and afirst translation vector for indicating a positional relation betweenbefore and after movement of the cameras.
 5. The apparatus of claim 4,wherein the positional relationship information between the stereocamera and the laser line scanner includes a second rotation matrix anda second translation matrix for indicating a positional relation betweena first camera included in the stereo camera and the laser line scanner.6. The apparatus of claim 5, wherein the set of converted coordinates isrepresented by the following equation:Xw(t)=R(t)[RsX(t)+ts]+t(t), wherein Xw(t) is the set of convertedcoordinates, wherein R(t) is the first rotation matrix, wherein Rs isthe second rotation matrix, wherein X(t) is a set of the coordinates,wherein ts is the second translation matrix, and wherein t(t) is thefirst translation matrix.
 7. A method for measuring a 3D position of anobject, comprising: obtaining frame-by-frame scan results for theobject, by a laser line scanner, using a line laser and a single camera;measuring position variation information of a stereo camera using imagesobtained from the stereo camera with respect to the object; andmeasuring 3D position information of the object based on the positionvariation information of the stereo camera and the scan results for theobject.
 8. The method of claim 7, wherein the measuring positionvariation information comprises: extracting feature points from imagescaptured by cameras included in the stereo camera and generating a firstfeature point list based on the feature points; extracting featurepoints from images captured by the cameras after the cameras have moved,generating a second feature point list based on the feature points, andrecognizing actual feature points based on the first feature point listand the second feature point list; and measuring the position variationinformation of the stereo camera based on variations in relativepositions of the actual feature points.
 9. The method of claim 8,wherein the measuring 3D position information comprises: converting thecoordinates of all the frames of the scan results into a coordinatesystem of a reference frame based on the position variation informationof the stereo camera and the positional relationship information betweenthe stereo camera and the laser line scanner; and obtaining a set ofconverted coordinates of all the frames as the 3D position informationof the object.
 10. The method of claim 9, wherein the position variationinformation of the stereo camera includes a first rotation matrix and afirst translation vector for indicating a positional relation betweenbefore and after movement of the cameras.
 11. The method of claim 10,wherein the positional relationship information between the stereocamera and the laser line scanner includes a second rotation matrix anda second translation matrix for indicating a positional relation betweena first camera included in the stereo camera and the laser line scanner.12. The method of claim 11, wherein the set of converted coordinates isrepresented by the following equation:Xw(t)=R(t)[RsX(t)+ts]+t(t) wherein Xw(t) is the set of convertedcoordinates, wherein R(t) is the first rotation matrix, wherein Rs isthe second rotation matrix, wherein X(t) is a set of the coordinates,wherein ts is the second translation matrix, and wherein t(t) is thefirst translation matrix.